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Agenda 


•  Motivation  for  “product” 

•  Product  Overview 

•  Topic  Details 

•  Workshop  Objectives 
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Motivation  for  “Product” 


“Risks  identified  are  rarely  realized,  risks  realized  were  rarely 

identified.” 

“All  unidentified  risks  are  accepted  risks.” 

•  Program  cost  and  schedule  overruns  can  be  traced  to  unrealistic  risk 
profile  at  program  inception. 

•  Aerospace  industry  have  experienced  critical  mission  failures  that 
could  have  been  avoided  if  the  failure  was  identified  as  a  risk  early  in 
the  program  life  cycle  and  properly  managed. 


U.S.  Space  Program 
Mission  Assurance 
Improvement 
Workshop 
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Unrealistic  Risk  Profile  at  Program  Inception 


Underestimation  of  risk  accounts  for  at  least  9%  of  space  systems  development  cost  growth. 


*  Booz  Allen  Hamilton.  (2002)  Moorman,  T.  S.,  et.al.,  Space  Systems  Development  Growth  Analysis,  Space  R&D  Industrial  Base, 
and  Conclusions  from  the  Space  Industrial  Base  Study 
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Unrealistic  Risk  Profile  at  Program  Inception 


Development  Growth  Causes 

(Stakeholder  Analysis) 

11% 


□  Requirements  Immaturity 

□  Requirements  Creep 

□  Programs  Budgeted  Too  Early 

■  Software  and  Integration  Underestimated 

□  Budget  Instability 

□  Competitive  Process  Over  optimism 


□  Program  Director  Turnover  &  Experience 


■  Inadequate  Pre-Acquisition  Planning  & 

Risk  Reduction _ 

□  Optimistic  and  Extrapolated  Estimates 

■  Lack  of  Systems  Engineers 


Other: 

-  Acquisition  Reform 

-  Program  Length 

-  Too  Many  KPPs 

-  No  Mgt  Reserve 

-  Program  Management 
Optimism 


□  All  Other  Causes 


Source:  77  surveys,  67  Organizations,  frequency  of  mention  versus  total  mentions 


The  highlighted  causes  are  directly  related  to  risk  identification  activities  at  program  inception. 
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Product  Overview 


A  guidance  document  was  chosen  to  capture  the  most 
common  risk  ID  barriers  and  to  offer  strategies  to  break  down 
the  barriers  to  improve  the  risk  ID  process. 

•  Overview  of  risk  identification  methods 

•  Risk  Identification  Barriers 

-  Describes  common  and  pervasive  barriers  that  act  to  obscure  risks 

•  Recommended  actions 

-  Describes  approaches  in  addressing  the  barriers 

•  Risk  identification  measures  of  completeness 

-  Check  list  for  assessing  the  quality  of  the  risk  ID  process 

•  Research  into  other  industries 

-  Summary  of  findings 
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<c 

Mission  Assurance  * -  1 

^  ■ 

Improvement 

Workshop  J 

U.S.  SPACE  PROGRAM  MAIW  |  DULLES,  VA  |  MAY  7 -8,  2014 


6 


Intended  Product  Use 


•  Who  is  the  target  audience? 

-  Risk  process  owners 

-  Risk/program  managers 

-  Customers 

•  How  should  /  could  it  be  used? 

-  Provide  guidance  to  improve  corporate  risk  identification  processes 

-  Helps  practitioners  avoid  pitfalls  in  conducting  risk  identification 

-  Provide  a  means  to  judge  effectiveness  of  their  risk  ID  process 

•  How  should  the  product  be  maintained? 

-  Product  should  be  reviewed  periodically  (every  few  years)  to  include  new 
barriers  or  methods  to  overcome/compensate  for  barriers. 
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Technical  Risk  Identification  Product  Traceability 


Deliverable  Requested 

Location  Covered  in  Product 

Risk  Identification  Method/Process 

Section  2:  Current  State  of  Risk  Identification  in  the  Aerospace  Industry 
Table  2:  Risk  Identification  Methods 

Recommended  program  resource 
engagement 

Section  4:  Barriers  to  Risk  Identification  and  Mitigation  Approaches 
(discussions  within  subsections  cover  recommendations  to  program 
resource  engagement) 

Section  5:  Recommended  Actions 

Risk  ID  checklist 

Section  5:  Recommended  Actions 

Risk  ID  quantitative  and/or 
qualitative  assessment 

Section  4:  Measure  of  Risk  Identification  Completeness 

Table  4:  Barrier  Mitigation  Scorecard 

Process,  tool,  format  for  risk 
summary/aggregation 

Section  3.2:  Barrier  2:  Artificial  Constraints  and  Biases 

Risk  ID  best  practices 

Section  4:  Barriers  to  Risk  Identification  and  Mitigation  Approaches 
(discussions  within  subsections  cover  recommendations  to  program 
resource  engagement) 

Section  5:  Recommended  Actions 

How  to  assess  contractor  technical 
risks  during  source  selection 

Section  4:  Barriers  to  Risk  Identification  and  Mitigation  Approaches 
(discussions  within  subsections  cover  recommendations  to  program 
resource  engagement) 

Risk  ID  for  oversight,  audit,  and 
review 

Section  4:  Measure  of  Risk  Identification  Completeness 

Table  4:  Barrier  Mitigation  Scorecard 

U.5.  Space  Program 

Mission  Assurance  * -  1 
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Topic  Details:  Risk  Identification  Methods 


•  Summary  of  Risk  Identification  Methods 


Risk  ID  Method 

Description 

Strengths 

Weaknesses 

Experiential  Methods 

Review  previous 
program  risks, 
issues,  and 
lessons  learned 

Review  of  risks  and  issues  identified  on  prior 
programs  of  similar  scope,  complexity,  and 
use  of  technologies  to  see  if  any  are 
applicable  to  the  current  program 

Leverages  relevant  knowledge  from 
similar  programs. 

May  not  include  risks  outside  of 
prior  programs'  experiences. 
Differences  between  programs  may 
not  be  understood. 

Checklists  and 
questionnaires 

Structured  method  to  identify  known  potential 
risk  areas  based  on  past  experience,  and  to 
have  responders  assess  the  applicability  of 
those  potential  risks  to  the  current  program. 

Leverages  institutional  and 
organizational  lessons  learned 

May  not  identify  risks  outside  of  the 
group’s  prior  experiences. 

Requires  organization  repository 
and  maintenance. 

Brainstorming 

Utilizes  social  interaction  to  enhance  the  risk 
identification  process.  It  requires  a  competent 
and  unbiased  facilitator  to  help  keep  the 
discussion  on  topic. 

Provides  a  structured  method  to 
leverage  the  knowledge  breadth  of  a 
diverse  group  of  experts 

Dominating  individuals  may  attempt 
to  push  their  ideas  onto  the  rest  of 
the  group,  and  weaker 
personalities  might  not  get  a 
chance  to  air  their  views. 

Only  as  good  as  the  experience 
breadth  of  the  group. 

Personal 
knowledge/ 
experience  of  risks 

Collect  risks  based  on  one  or  more 
individual’s  personal  knowledge  and 
expertise. 

Beneficial  within  each  individual’s 
experience  range 

Individuals  generally  lack  the  full 
breadth  of  understanding  of  the 
entire  program  and  may 
inadvertently  filter  out  credible  risks 
from  their  identification  process. 

Analytical:  Methods  based  on 
analysis  of  data. 


U.S.  Space  Program 
Mission  Assurance 
Improvement 
Workshop 


Experiential:  Methods  based  on 
personal  experience  and 
knowledge. 


Analytical 

Key  Performance 
Parameters 
(KPPs)  - 
technical, 
programmatic 

Review  of  the  KPPs  to  identify  the  specific 
risks  to  achieving  the  key  program 
objectives. 

Provides  risk  identification  that  is 
targeted  on  the  design's  ability  to 
meet  the  program's  KPPs 

Assumes  the  Program's  identified 
KPPs  fully  represent  the 
parameters  that  best  represent  the 
required  system  performance. 

Review  Project 

Work  Breakdown 
Structure  (WBS) 

A  critical  review  of  the  WBS  can  expose  risks 
inherent  in  the  interdependency  of  the  project 
work 

Provides  a  structured  approach  for 
risk  identification  in  the  context  of 
how  the  program’s  work  is 
structured,  including  entities  external 
to  the  program  (suppliers, 
teammates,  governmental  entities, 
etc.) 

Risk  identification  using  the  WBS  is 
only  as  good  as  the  WBS  itself,  and 
the  expertise  of  the  risk  identifiers 
reviewing  the  WBS. 

Risk  Breakdown 
Structure 

Risks  are  stated  and  assessed  at  each  level 
of  architectural  assembly:  system, 
subsystem,  unit,  component  and  part. 

Higher  level  risk  assessments  are  informed 
largely  by  historical  data.  Middle  level  risks 
also  include  the  risk  of  interface  and 
interaction.  Component  and  part  level  risks 
are  only  assessed  for  very  high  unit-level 
risks. 

Comprehensive,  structured,  and 
intuitive  for  the  reviewer. 

Aggregate  risks  include  the 
probabilistic  sum  of  all  of  the 
constituent  elements. 

Aggregation  is  subjective,  and 
typically  not  statistical  or 
mathematical  -  resulting  in 
decreased  confidence. 

Low  aggregate  risks  may  mask 
high  concentrations  of  risk  in 
certain  components  or  parts. 
Effective  mitigation  is  sometimes 
best  performed  at  a  different  level 
than  the  level  being  reviewed. 

Inception  Risk 
Standardization 

Each  program  assesses  and  dispositions  a 
list  of  pre-defined  standard  risks  based  on 
the  experience  and  data  collected  from 
historical  programs  and  missions 

This  method  requires  programs  to 
assess  likely  risks  which  may  be 
overlooked. 

Pre-defined  standardized  risk  lists 
are  not  likely  to  be  insightful  to 
mission  and  program  peculiar  risks. 

Review 

Requirements, 

riPdinn  nnmmpntQ 

Review  of  these  documents  can  reveal 
perceived  gaps  in  the  design,  or  over- 

rnnctraintc  that  rnnlH  aH\/Pr<;p|\/  affprt  Hpcinn 

Provides  a  structured  approach  for 
risk  identification  in  the  context  of  the 

nrnnram'g  rpnnirpmpntc;  anri  rlp^inn 

Risk  identification  using  the 
requirements  and  design 

Hnrnmpntatinn  k  nnlu  ac  nnnH  ag _ 

U.S.  SPACE  PROGRAM  MAIW  |  DULLES,  VA  |  MAY  7 -8,  2014 


9 


Topic  Details:  Risk  Identification  Barriers 


•  Major  Risk  Identification  Barriers 

-  Barrier  1:  Over-Reliance  on  a  Single  Method 

-  Barrier  2:  Artificial  Constraints  and  Biases 

-  Barrier  3:  Dismissing  a  Risk  as  a  Normal  Program  Challenge 

-  Barrier  4:  Compliance  Reliance 

-  Barrier  5:  Program  Acquisition  Attributes 

-  Barrier  6:  Scope  Boundaries 

-  Barrier  7;  Normalization  of  Deviance 

“There  are  many  barriers  to  risk  identification  commonly  encountered  within 
the  space  community.  Some  of  these  barriers  are  intrinsic  to  an 
organization’s  processes  and  others  are  inherent  in  human  psychology. 
These  barriers  impact  the  ability  of  customers,  contractors,  and  risk 
practitioners  to  effectively  identify  risks.” 


U.5.  Space  Program 
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Topic  Details:  Measure  of  Risk  ID  Completeness 


•  Utilization  of  Barrier  Mitigation  Scorecard 


Risk  ID  Area 

Barrier  Mitigation  Assessment  Criteria 

Assessment 

General:  Barriers 

1  None  of  the  identified  barriers  have  been  addressed 

3  Some  of  the  identified  barriers  have  been  addressed 

6  The  most  significant  barriers  have  been  addressed 

9  All  barriers  have  been  addressed,  or  are  not  applicable 

Barrier  1:  Over- 
Reliance  on  Personal 
Experience 

1  The  program  has  no  method  other  than  personal  experience  to  identify  risk 

3  The  program  has  access  to  many  of  the  risk  ID  methods,,  but  choice  of  method  is  at 
individual  discretion 

6  The  program  has  access  to  many  of  the  risk  ID  methods,  and  provides  guidance  on  the 
appropriate  usage 

9  The  program  requires  application  of  multiple  risk  ID  methods 

Barrier  2:  Application 
of  Artificial  Constraints 
and  Biases 

1  The  program  has  explicitly  established  an  artificial  barrier  (e.g.,  a  Top  N  risk  list) 

4  The  program  has  no  risks  above  a  certain  level  and  does  not  challenge  this 

9  Risk  identification  process  does  not  have  limits  placed  on  the  number  and/or  magnitude 
of  risks.  The  program  actively  challenges  assumptions  that  may  constrain  risk 
identification,  such  as  a  low  risk  program  having  no  medium  or  high  risks. 

Barrier  6:  Separating 
Risk  from  Planned 
Work  in  Development 
Programs 


1  Risk  ID  is  completely  dependent  on  individual  perspective,  and  risks  are  inconsistent 
with  respect  to  future  development 

3  Development  risk  is  filtered  during  risk  review  to  consistently  discriminate  between 
risks  within  and  outside  of  plan 

6  Development  risks  are  consistently  identified  by  using  multiple  risk  ID  methods 

9  Risk  ID  specifically  solicits  development  risk  by  using  multiple  risk  ID  methods  during 
all  program  development  activity 


Barrier  7:  Risks 
Confused  with  Issues 


1  No  consistent  and  integrated  risk  and  issue  management  training  of  program  team 
3  Risk  and  issue  training  in  place  and  conducted  regularly  with  program  personnel 
6  Risk  Manager  actively  engages  when  candidate  risks  are  identified  to  ensure  issues  are 
identified  and  immediately  elevated 


U.S.  Space  Program 
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Topic  Details:  Recommended  Actions 

•  A  set  of  recommended  actions  for  organizations  to  implement 
improved  technical  risk  identification  at  program  inception 


ID 

Recommended  Action 

Barrier 

Mitigated 

Action  Taken  By: 

1 

Review  list  of  risk  identification  methods  and  provide  processes,  tools, 
templates,  and  training  to  program  risk  managers 

Barrier  1 

Risk  Management 
Process  Owner 

2 

Plan  to  use  several  methods  of  risk  identification  at  each  program 
milestone  and  document  it  in  the  risk  management  plan  (or  equivalent). 

Barrier  1 

Program  Risk  Manager 

3 

Require  at  least  two  risk  identification  methods  for  each  program  milestone 
or  event,  using  at  least  one  experiential  method  and  one  analytical  method. 

Barrier  1 

Risk  Management 
Process  Owner  and 
Program  Risk  Manager 

4 

Monitor  risk  identification  process  for  barriers  and  implementation  of 
recommended  solutions.  This  independent  review  should  focus  on  the 
process,  and  be  separate  from  the  review  of  risks. 

Barriers  1-7 

Risk  Management 
Process  Owner 

5 

Streamline  process  for  candidate  risks  to  minimize  the  investment  in  initial 
capture  and  review.  If  not  accepted,  candidate  risks  should  be  revisited 
periodically  to  assure  that  their  state  has  not  change. 

Barriers  2, 

7 

Risk  Management 
Process  Owner, 

Program  Risk  Manager 

6 

Streamline  process  for  risk  monitoring.  Do  not  reject  or  close  risks  that 
have  not  been  realized  or  completely  mitigated,  but  require  only  periodic 
review. 

Barriers  2, 

7 

Risk  Management 
Process  Owner 

7 

Normalize  risks  to  the  unit  or  interface  level.  Consolidate  risks  that  are 
narrower  than  unit  function,  and  separate  large  aggregate  risks. 

Barrier  2 

Program  Risk  Manager 

8 

Add  risk  identification  as  a  discrete  activity  throughout  the  life  cycle 
(program  milestones,  design/readiness/peer  reviews,  of  technical 
products  (code  reviews,  modeling  and  simulation,  rehearsals,  and 

Barriers  1-4 

Program  Risk  Manager; 
Risk  Management 
Process  Owner 

U.5.  Space  Program 

Mission  Assurance 

Improvement 

Workshop  J 

brokers,  and  key  technology  suppliers. 

21 

Candidate  risks  should  be  processed  prior  to  Risk  Management  review. 
Filtering  may  include  screening,  combining,  interpreting,  equalizing,  and  re¬ 
framing  original  ideas.  This  initial  risk  list  has  substantial  value-added  and 
is  much  more  useful/actionable  than  raw  list  of  candidate  issues 

Barrier  2,  4, 

7 

Program  Risk  Manager 
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Topic  Details:  Other  Industries 

•  Industries  Researched: 

-  Auto  Insurance 

-  Supply  Chain 

-  Highway  Transportation 

-  Medical  Industry 

-  Nuclear  Reactor  Safety 

-  Oil  and  Gas  Industry 

-  Commercial  Launch  and  Space  Insurance 

-  US  Government  Intelligence  Analysis 

•  The  industry  research  reinforced  common  risk  identification  practices 

-  Checklists  and  databases  are  commonly  accepted  tools  for  risk  identification 

-  Cognitive  barriers  exist  and  need  recognition  and  time  to  overcome 

-  Industry  working  groups  and  communities  of  practice  are  valuable  resources 
for  risk  identification 
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Workshop  Objectives  and  Accomplishments 


•  Incoming  objectives  of  workshop 

-  Obtain  consensus  on  the  adjudicated  SME 
Comments,  resolve  any  non-concurrences 

-  Clarify  and  disposition  all  remaining  (To  Be 
Resolved)  SME  Comments 

-  Incorporate  any  new  SME  changes  and 
recommendations  into  final  Silver  Version  or 
into  Gold  Version 

•  Workshop  Accomplishments 

-  Adjudicated  all  outstanding  SME  comments 

-  Incorporated  SME-motivated  changes 

-  Performed  document  clean-up 


Total  SME  Comments  99 

Status  entering  the  Workshop 


■  Accept 

■  Reject 

■  Modified  with 
Changes 
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Future  Work 


•  2014  MAIW  product:  Risk  Identification  at  Program  Inception 
guidance 

-  Recommend  introduction  of  this  product  to  PM  I,  INCOSE  communities 

-  Recommend  incorporation  of  this  guidance  document  into  the  existing 
Aerospace  family  of  risk  management  TORs  and  handbooks. 

•  Future  MAIW  topics 

-  Look  beyond  risk  identification  to  include  risk  management  process 
improvement 

•  Several  barriers  cannot  be  overcome  without  the  modification  of  the 
overall  risk  management  process 

-  Approaches  to  changing  the  cultural  perception  of  risk  management 

•  Program  managers  may  not  perceive  an  immediate  value  to  a  formal 
risk  management  process. 

•  Risk  management  resources  may  be  challenged  in  a  budget- 
constrained  environment 


U.5.  Space  Program 

J- 

Mission  Assurance  * -  1 

^  ■ 

Improvement 

Workshop  J 

U.S.  SPACE  PROGRAM  MAIW  |  DULLES,  VA  |  MAY  7 -8,  2014 


15 


Core  Team 


Team  Introductions 


SMEs 


Company 

Participant 

Company 

Participant 

The  Aerospace 
Corporation 

Andrew  Hsu  (co-lead) 

Amy  Weir  (co-lead) 

The  Aerospace 
Corporation 

Anh  Dang 

Sergio  Guarro 

Ball  Aerospace  & 
Technologies  Corp 

Bill  Frazier  (co-lead) 

Ball  Aerospace  & 
Technologies  Corp 

David  Pinkley 

The  Boeing  Company 

Robert  Ellsworth 

Booz  Allen  Hamilton 

Brian  Weir 

Lockheed  Martin 
Corporation 

Jerome  Sobetski 

Aerojet  Rocketdyne 

Alexis  Burkevics 

Northrop  Grumman 
Aerospace  Systems 

Debra  Olejniczak 

Northrop  Grumman 
Aerospace  Systems 

Dennis  Rubien 

LaKeisha  Souter 

Orbital 

John  McBride  (co-lead) 

Orbital 

Ben  Hoang 

Jaclyn  Decker 

Raytheon  Space  and 
Airborne  Systems 

Robert  Jennings 

SMC 

Dave  Davis 
(Government) 

SSL 

Larry  Rubin 
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Final  Thoughts 

Two  risks  walked  into  a  bar  and  one  was  kicked  out  because  he 
became  an  issue. 

A  risk,  a  concern,  and  an  issue  were  heading  to  a  bar. 

The  risk  couldn’t  get  in  because  he  was  using  a  fake  ID. 

The  concern  argued  with  the  bouncer  and  was  man-handled. 
The  issue  snuck  in  unnoticed. 

Why  did  the  risk  cross  the  road?  To  get  to  greener  pastures. 

Why  did  the  issue  cross  the  road?  To  get  to  the  root  cause. 

Knock,  knock.  Who’s  there?  Concern.  Concern  who?  Concerns  you 
if  you  don’t  mitigate  this  risk! 

If  a  risk  turns  into  an  issue  and  no  one  is  around  to  see  it,  will  it  still 
cause  a  failure? 
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